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Abstract 

We investigate how undecidability enters into computations of classical physical systems and 
contributes to the increase of entropy and loss of information. In actual computation with finite 
bit of information capacity we accept inconsistency to avoid undecidability, which in turn affects 
entropy of the system. We apply the Shannon entropy to the discretized Liouvillian system. It is 
shown that for any finite bit of information capacity information is always lost or the entropy always 
increases for the probability density following Hamiltonian dynamics, both in time forward and time 
backward direction, thus showing information theoretical version of second law of thermodynamics. 
This is due to the finiteness of information capacity and incompressibility of probability distribution 
in Liouville's equation. 
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In 1931, two famous theorems jj are presented by K. Godel. The first theorem states that 
any axiomatic system that is strong enough to express natural numbers contains undecidable 
statements, which can be neither be proved or disproved within that system. The second 
theorem states that no consistent system can be used to prove its own consistency. 

After Godel's work, interesting variation of these theorems appeared, notably in computer 
science and algorithmic information theory. In 1936, A. Turing proved that the halting 
problem, the question of whether or not a Turing machine halts on a given program, is 
undecidable Beginning in late 1960s, G. Chaitin showed that in a formal system with 
n bits of axioms it is impossible to prove that a particular binary string is of Kolmogorov 
complexity greater than n + c [3|. Put it roughly, he states that for arbitrary n bit number 
it is not possible to make a smaller size program which prints that number. It is also shown 
that majority of n bit numbers are maximally complex, i.e. the size of program that can 
print that number is 0(n). 

How undecidability enters into physics is also researched and undecidable problems in 
physics are presented 0, . In this article we consider simpler undecidable problems when 
one tries to simulate classical physical system by a computer. Many physical systems are 
expressed with continuous real numbers, but computers can deal with only finite bits of 
information. If one ask to a computer "What is the decimal expression of a real number 
p?" , then the computer tries to answers 

p = Xq.X^iX-2%-3X-4: , (1) 

(in Eq. flTJ), XQ.X-1X-2X-.3X-4... denotes the decimal digits.) but in almost all cases the 
digits make a non-repeating infinite sequence. Computing higher order of digits can be 
extremely hard and whether or not there exists a simple algorithm to calculate p is in 
general undecidable problem, i.e. the computer never halts or it cannot decide the exact 
answer in finite time. To solve this problem, usually people truncate or round off the answer 
up to some finite digits and write 

P = Xo-X_iX_2^-3^-4- (2) 

By truncation or roundoff, Eq. (j2J) becomes a mathematically false statement. The equation 
becomes inconsistent by avoiding undecidability, like Godel's two theorems. In numerical 
simulation this inconsistency is called as numerical error or round off error. This inconsis- 



tency, or roundoff errors, in turn contribute the entropy of the result, by reducing significant 
digits of the result. 

According to Shannon 6j, the entropy of the discrete system with n event probability 
Pi, P 2 ..., P n is defined as 

n 

H = -J2Pi^Pi- (3) 

i=i 

From now on we take the base of the logarithm as 2. This entropy is a measure of uncertainty 
or degree of freedom of the system or the information capacity of the system. H in Eq. ([3]) 
is always a positive quantity since < Pi < 1. When the entropy decreases, we call that the 
uncertainty is reduced or information is gained. For example, if we have an unknown digit 
X which can be either or 1 with probability 1/2 each, we have 1 bit of entropy by Eq. (J3j). 
The system has one bit of uncertainty or one bit of degree of freedom or the ability to store 
one bit of information (1 bit information capacity). If the unknown digit X is identified as 
0, then probability of being is 1 and by Eq. ([3]) H becomes zero bit. The uncertainty or 
degree of freedom is decreased by one bit, and we gain 1 bit of information and no degree 
of freedom in this system. 

Now we consider entropy appearing in the computation of classical physical systems. 
The information loss and entropy increase for chaotic systems [[]] and generalized Liouville's 



systems 
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are already studied, and as a sources of information loss discarding of information, 
interaction with environment or coarse graining are pointed out. In this article we restrict 
our attention to the classical Hamiltonian systems which follow the Liouville's equation. 
We disretize the system and see how the entropy changes with time evolution. It is shown 
that the calculation of Shannon entropy for the discretized system naturally separate the 
information capacity of the system and Kullback information, and the information is always 
same or lost for any finite discretization of probability density following the Hamiltonian 
dynamics. 

Consider the probability distribution function p(p, q, t) of a particle in the phase space, 
which satisfies Liouville's equation. Suppose that p has compact support and that support is 
contained by finite size box Q with volume Qq. For numerical computation we discretize the 
phase space Q by N number of uniform box shaped cells, with each cell has the volume Qq/N. 
Let us denote the cells as Ci, C 2 , Cat, and we approximate the probability distribution 
p(p, q, t) inside the cell Ci as Pi, which is the mean value of p(p, q, t) inside the cell Cj. 



This is the place the small inconsistency enters. Since we cannot describe the probability 
distribution function with infinite precision, we replace the distribution function p(p, q, t) 
with a mean value Pi inside the cell (coarse-graining over the cell). 

The discretized probability density PiS satisfy the relation Y2iLiPi(^o/N) = 1 and < 
Pi < N/Q . Suppose that we have initial condition pf\p^\ ■■^pff for every cell Cj at time 
t = 0. The probability that the particle is in the cell Cj is p^ (Qo/N), and the entropy 
of the discretized system at t = is given by 

N 

Hi0) = -^(W^fM) 

i=l 

N N 

= - 5>; o) (fi o /A0 \og(pi 0) Q ) - Y,P?\n /N) log(l/iV) 

i=l i=l 
N , (0) n 

= ~YJf\^im fog ( (^yj + ^ ( 4 ) 

The entropy if'- -' has maximum value logiV for uniform distribution, i.e. when all pf = 
l/^o, and minimum value when pf* = N/Qq for one specific i and V$L% = fo r an other 
js. So we have 

N , (0) n 

< = _ J2p?\n /N) log (^<^yj + logiV < logiV 

N , (0) \ 

- log iV < - g pf (flo/JV) log ^ A_ J < o (5) 

The meaning of terms in Eq. (J4]) are following. The maximum entropy log A" is the informa- 
tion capacity or the number of bits allowed for us to describe the location of a particle in 
phase space. For uniform distribution, we have no information of the location of the particle 
and all allowed bits are remain unknown. When a particle is in the one cell with probability 
1, all unknown bits are fixed and uncertainty is 0. In this case we get maximum informa- 
tion within allowed information capacity. The term — YliLiPi^i^o/N) log(p^ /(1/fio)) m 
relation (jSJ) is called Kullback-Leibler divergence [7| or relative entropy with respect to the 
uniform distribution. We see that this term is always non-positive, so this is actually the 
information of particle location we get from the system. Note that this term converges to 
the integral — f dVL p(p, q, 0) log(p(p, q, 0)/(l/f2 )) as N becomes large, so for larger N the 
information we get is more dependent on the integrability of p(p, q, 0) and less dependent 
on the number of discrete cell N. 
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Next we consider the time evolution of probability density in this discretized system. 
Since it is a classical Hamiltonian system, as time changes the probability density moves 
like incompressible fluid in phase space, i.e. if one follows the time evolution of a point in 
phase space, the density at the representation point remains constant and the volume of the 
neighborhood at the point is conserved. The original cell C^s deform, but the discretized 
probability density inside the deformed cell is still pf\ Let us denote the deformed cells 
after one discrete time step as C> s. In general we cannot track down the deformed cells 
with infinite precision. It is undecidable problem [5j . In practice we see the system with our 
discretized fixed cells of CjS, and the new mean discretized probability density p^ which 
is averaged over Qs. The original cell Cj may contain many deformed cell Cj s, and each 
overlap between Cj 1 ^ with Cj contributes to the new mean probability density pp (see 
figure [1]) . We have 

N 



n=l 



where a im is given by 



volume of Cj n Cm 
volume of G 



(7) 



This averaging is the place where small inconsistency enters, to avoid undecidability due to 
the finite information capacity. In Eq. (|7j) and from the fact U^ =1 Cm' ) = l)f =1 Ci = f2 we 
have the relation 

N N 

< a im < 1, E a im = E aim = 1- 

m=l i=l 

After one time step, the new entropy H^ 1 ' looking through C^s is 

N / 

Hil) = ~ log (f/n ) + logN 

N N /\^ N (°)\ 

= - E E ^ wn) log ( ^^mp m + log K (9) 

i=l m =l ^ / / 

Since the function /(x) = xlog(Ax) with A > is a convex function and the convex function 
satisfies Jensen's inequality 

fC^2aiXi)<^2aif(xi) for all a, t > 0, (10) 
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FIG. 1: The new discretized probability density p\ . In the left figure, each square shaped cells 
has discretized probability density p^s. (i = 1,..,4) After one discrete time step the cells are 
deformed (shown as dashed parallelograms). The new discretized probability density in C2 cell 
(the square with thick line in the right figure) is obtained by averaging the portions of probability 
densities moved into the C2 cell. 



the first term in RHS of Eq. <Q is (with Eq. §_ 

N N , s—\N 



m=l a imP 




(0) 



N N / (0) 

i=l m=l 



N 



m=l 



D (0) 



From ( [IT]) and Eq. we have 



H (0) < #d) 



(12) 



i.e. the information is always lost or the entropy always increases. Since the property 
that the probability density distribution moves like incompressible fluid under Liouville's 
equation does not change when it is evolved backward in time, we can do the time evolution 
of p/°^s backward and get the same result < if( -1 ) where H^~ 1 ^ is the entropy in one 
discrete time backward. So we have the information theoretical version of the second law 
of thermodynamics, i.e. the information is irreversibly lost during the time evolution of the 
classical Hamiltonian system. 

There are two key properties which make the irreversible information loss in our case. 
One is the fixed finite resolution of the system, which forces us to take the mean value 
of the probability density over the cell. Second is the incompressibility of the probability 
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distribution during time evolution. Without any one of properties the relation ffl2|) will not 
hold. 

In conclusion, it is shown how the undecidability enters into classical physical system 
simulation and contribute to the information loss. Due to the fmiteness of bits we are using, 
we have to choose between undecidability and inconsistency. When we choose inconsis- 
tency, it affects the uncertainty of the system. When we examine the time evolution of the 
probability distribution in the Liouville's equation with finite fixed information capacity, 
the information is always lost in both directions of time or entropy always increases. As 
Jaynes said [h], this is one way of looking statistical mechanics law on the basis of the 
information one can get. 

The author would like to thank Moo Young Choi, Seunghwan Kim and Gonzalo Ordonez 
for helpful comments. 
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